Chapter 1 - Introduction to Language Models

Phi-3

The first step is to load our model onto the GPU for faster inference. Note that we load the model and tokenizer separately (although that isn’t always necessary).

from transformers import AutoModelForCausalLM, AutoTokenizer

# Load model and tokenizer
model = AutoModelForCausalLM.from_pretrained(
    "microsoft/Phi-3-mini-4k-instruct",
    device_map="cuda",
    torch_dtype="auto",
    trust_remote_code=False,
    attn_implementation="flash_attention_2",
)
tokenizer = AutoTokenizer.from_pretrained("microsoft/Phi-3-mini-4k-instruct")
`torch_dtype` is deprecated! Use `dtype` instead!

Although we can now use the model and tokenizer directly, it’s much easier to wrap it in a pipeline object:

from transformers import pipeline

# Create a pipeline
generator = pipeline(
    "text-generation",
    model=model,
    tokenizer=tokenizer,
    return_full_text=False,
    max_new_tokens=1000,
    do_sample=False
)
Device set to use cuda
The following generation flags are not valid and may be ignored: ['temperature']. Set `TRANSFORMERS_VERBOSITY=info` for more details.

Finally, we create our prompt as a user and give it to the model:

# The prompt (user input / query)
messages = [
    {"role": "user", "content": "Write a funny joke about herons."} ]

# Generate output
output = generator(messages)
print(output[0]["generated_text"])
 Why don't herons ever play hide and seek? Because good luck hiding when you're always standing out with your long legs and neck!
Back to top